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As a prototypical self-organization in the system with orbital degeneracy, we theoretically investigate 
trimer formation on a triangular lattice, as observed in LiV02. From the analysis of an effective spin-orbital 
coupled model in the strong correlation limit, we show that the previously-proposed orbital-ordered trimer 
state is not the lowest-energy state for a finite Hund's-rule coupling. Instead, exploring the ground state 
in a wide range of parameters for a multiorbital Hubbard model, we find an instability toward a different 
orbital-ordered trimer state in the intermediately correlated regime in the presence of trigonal crystal field. 
The trimer phase appears in the competing region among a paramagnetic metal, band insulator, and Mott 
insulator. The underlying mechanism is nesting instability of the Fermi surface by a synergetic effect of 
Coulomb interactions and trigonal-field splitting. The results are compared with experiments in triangulax- 
lattice compounds, LiVX2 {X=0, S, Se) and NaV02. 

KEYWORDS: cluster formation, trimer, multiorbital Hubbard model, triangular lattice, orbital order, metal- 
insulator transition 
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Orbital degree of freedom plays a decisive role in many 
transition metal compounds. Orbital ordering and 
its fluctuation affect magnetic, transport, and structural 
properties in a complicated manner through the cou- 
pling to spin, charge, and lattice degrees of freedom. One 
of the characteristic phenomena, particularly observed 
in geometrically-frustrated systems, is spontaneous for- 
mation of clusters. It is a structural phase transition 
to form 'molecules' of transition metal atoms by peri- 
odic modulation of the lattice structure. For example, 
CuIr2S4 shows a formation of octumers^^ and AIV2O4 
shows heptamers'"''' on their pyrochlore networks of Ir or 
V, and the importance of d-electron orbitals was theo- 
retically pointed out.^'''-* The spontaneous formation of 
'molecules' has attracted increasing interest as emergent 
physics in the systems with multiple degrees of freedom. 

A prototypical example of such clusters is a trimer in a 
quasi- two-dimensional compound LiV02.^'^^ In this sys- 
tem, each V'^"'" cation has two Sd electrons in threefold 
t2g orbitals on average, and forms triangular lattices. The 
compound exhibits a first-order structural phase transi- 
tion at Tc ~ 500 K, accompanied by a sudden drop of 
the magnetic susceptibility.'^' ^"^^ Below Tc, V-V bonds 
are periodically modulated in each layer to form V3 
trimers.^"^^^ The trimer formation was first interpreted 
by a charge-density-wave instability associated with the 
V clustering.^' ^'^^ A spin-Peierls type scenario was also 
considered to account for the spin-singlet nature below 
Tc}'^^ Later, Pen et al. pointed out a crucial role of or- 
bital ordering: It was claimed that LiV02 is a Mott 
insulator and the trimers are stabilized as the S = 1 
spin-singlet objects under a particular ordering of the 
t^g orbitals. 

Experimentally, however, it is not clear to what ex- 
tent the electron correlation plays an important role in 
LiV02. The spin gap in the low-temperature(T) trimer 
phase was estimated to be Ag 1600 K,^'*) which is com- 
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parable to the charge gap estimated from T dependence 
of the resistivity, Ac ^ 0.14 eV.^^^ In addition, the Curie- 
Weiss temperature estimated from the magnetic suscep- 
tibility for T > Tc is -1500 K,!^) ^hose magnitude 
is also comparable to Ac. These facts clearly contradict 
with the local moment picture under the strong correla- 
tion. Recently, it was also shown that the substitution of 
O by S or Se makes the system metallic. This suggests 
that LiV02 is located in the vicinity of correlation-driven 
metal-insulator transition. It is highly nontrivial how the 
itinerant tendency is compatible with the S = 1 local mo- 
ment formation expected for the strong electron correla- 
tion. Hence, the origin of the trimer formation in LiV02 
is still in dispute. It is desired to revisit the problem the- 
oretically for clarifying the role of electron correlation 
and orbital degree of freedom. 

In this Letter, we address this issue by carefully exam- 
ining various orbital and spin orderings in a wide range of 
electron correlation. In particular, revisiting the strong 
correlation picture, we show that the system is likely to 
exhibit a four-sublattice spin and orbital ordering in the 
ground state, not the previously-proposed trimer state. 
We, however, identify a different orbital-ordered trimer 
state in the intermediately correlated region on the verge 
of metal-insulator transition, where the Coulomb inter- 
actions work cooperatively with the trigonal-field split- 
ting. The results provide underlying mechanism in the 
self-organization in orbital degenerate systems close to 
metal-insulator transition. 

We start with a multiorbital Hubbard model for the 
threefold t2g orbitals. The Hamiltonian is given as 

^ = -EEE^S'(4.c.^^+H.c.) 

{ij) afS T 

+ E ^Ua[ia'P'c\^r4pT'^^fi'r'Cia'r, (1) 

i OipOi' P' rr' 

where the first term is the electron hopping between the 
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nearest-neighbor sites (ij) on a triangular lattice; a, /3 
denote the three t2g orbitals, d^y, dyz, or dzx', t,t' de- 
note the spins, f or 4,. The second term represents the on- 
site Coulomb interactions, for which we use the standard 
parametrizations, UajSa'P' ~ U'Saa'Spi^' + Jh('^q;3"^^q' + 
SajsSa'is') and U = U' + 2Jh- Following the previous 
study, we here consider the overlap integrals of cr-bond 
orbitals only and take them as the energy unit, t„ = 1. 
We fix the electron density at n = Y^ia{nia)/N = 2, i.e., 
two electrons per site on average (iV is the number of 
sites). 

First, we consider the strong correlation limit of the 
model in Eq. (1). The second order perturbation in t„ /U 
gives an effective spin-orbital Hamiltonian^-* in the form 



(ij) 



AF 



(2) 



where h, 



(y) 



o-AF 



= (A 



BSi ■ Sj){ni^i^ij-^nj^(^ij^ -\- 

Here Si is the S = 1 spin operator at site i, a(ij) is the 
orbital which has the cr-bond overlap between the sites 
i and j, and fiiai^ij) = 1 — nia(ij). The coefficients are 
given as J = tl/U, A = (1 - r;)/(l - 877), B = 77/(1 -3??), 
C = (1 + 77)/(l + 277), and 77 = Jh/U. The same effective 
model was studied for LiVOa:^^) It was shown that an 
S = 1 spin-singlet state with trimer-type orbital ordering 
[Fig. 1(a)] has the same ground-state energy as an an- 
tiferromagnetic (AF) state with a 'square-type' orbital 
ordering [Fig. 1(b)] in the limit of r; = 0, i.e., U = U' 
and Jfj = 0. However, it was not clarified how a finite 
Hund's-rulc coupling modifies the results, and moreover, 
whether there are other competing states. 

In order to search the ground state of the model in 
Eq. (2) in an unbiased way, we performed Monte Carlo 
(MC) simulation at low T for the classical counterpart of 
the model, namely, by replacing the S = I operators by 
the classical vectors with unit length. The results indi- 
cate that, for 7; > 0, a four-sublattice ferrimagnetic state 
shown in Fig. 1(c) is selected as the lowest-energy state 
as T ^ 0. 

On the basis of the MC result, we compare the ground- 
state energy of the ferrimagnetic state with other sev- 
eral typical states, as plotted in Fig. 2. Here we con- 
sider four states in Figs. l(a)-(d). For the states (b)-(d), 
the ground-state energies are analytically obtained as 
Esq-AF = -4.CJ, Ei,,,i = -{2A + B + 2C)J, ^120- af = 
— {2A — B + 3C/2) J, respectively, by treating the model 
(2) at the classical level. For the trimer state (a), as- 
suming the 5 = 1 spin-singlet state in each trimer, 
namely, {Si ■ Sj) = —1 for the intra-trimer bonds, and 
neglecting the intcr-trimer spin correlations, we obtain 
^-trimer = —{2 A + 2C)J. Tlic Comparison indicates that 
the four-sublattice ferrimagnetic state gives the lowest 
energy for 77 > 0, consistent with the MC search. The 
energy is even lower than the previously-proposed trimer 
state. Efcm will be lowered when considering quantum 
fluctuations beyond the classical level, such as the spin 
wave contribution. Therefore, our results strongly sug- 
gest that the trimer state proposed in Ref.^^^ is not the 




Fig. 1. (Color online). Schematic pictures of the spin-orbital or- 
dered states: (a) orbital trimer state proposed in Ref. 15, (b) AF 
state with square- type orbital ordering, (c) ferrimagnetic state with 
four-sublattice spin-orbital ordering, (d)-(f) Hartree-Fock solutions 
appearing in the phase diagram in Fig. 3(a). Only dominant or- 
bitals are drawn by lobes in each figure. Arrows represent spins. 
In (a), the shaded triangles denote the trimers with spin-singlet 
formation. 
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Fig. 2. (Color online). Ground-state energy of the model (2) for 
the four states in Figs. l(a)-(d) as a function of 77 = Jn/U. See 
text for details. 



ground state in the strong correlation limit, and is likely 
taken over by the four-sublattice ferrimagnetic state in 
Fig. 1(c). 

The above considerations in the large-?/ limit lead us 
to go back to the multiorbital Hubbard model given by 
Eq. (1) and explore another possibility for the trimer 
formation. In the following analysis, we extend the model 
by including the trigonal distortion of VOg octahedra 
which is inherent in the layered materials. That is, we 
consider an additional term to Eq. (1) given by 

nn^^y.Y.T.4arC.,r, (3) 



2 ^ 



which splits the levels into aig singlet and e'g dou- 
blet by 3D/2. The sign and magnitude of D is strongly 
dependent on the detailed band structure including the 
t2g-£g hybridization. ^^-^ In the following calculations, we 
consider the case of Z? > which lowers the aig level. 

To map out the ground-state phase diagram in a wide 
parameter region, we employ the Hartree-Fock approxi- 
mation. In the calculations, we take the 12-site unit cell 
in the form shown in Fig. 1, and consider 24 x 24 array of 
the unit cell with appropriate boundary conditions. This 
enables to incorporate both three- and four-sublattice 
orders, such as the trimer and ferrimagnetic states. 
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Fig. 3. (Color online), (a) Ground-state phase diagram for the 
multiorbital Hubburd model at the electron density n = 2 ob- 
tained by the Hartree-Fock calculation. The thick curve represents 
the boundary between magnetic and nonmagnetic solutions (the 
upper left is magnetic). The dotted curve represents the metal- 
insulator transition line. Spin and orbital patterns for the states 
b-f are drawn in Figs. l(b)-(f), respectively, (b) A schematic pic- 
ture of the spin and orbital state in the trimer phase, (c) Total spin 
(S^) (square) and energy gap (circle) as a function of (7 at D = 0.7 
(open) and 1.1 (closed). 



Figure 3(a) shows the representative resuh for the 
ground-state phase diagram as functions of U and D. 
We take r] = 0.1, i.e., U' = 0.8U and Jh = O.IU. When 
[/ = 0, the system exhibits a metal-insulator transition 
from paramagnetic metal to band insulator at Z? = 2 by 
the trigonal-field splitting of aig and e'g bands. On the 
other hand, at D = 0, the system is insulating for all 
U > because of the perfect nesting of the Fermi sur- 
face. In the large-C/ region, we obtain the four-sublattice 
orbital-ordered ferrimagnetic state [Fig. 1(c)], in agree- 
ment with the above result for the large-f7 effective model 
in Eq. (2). We, however, do not find any trimer-type 
threefold ordering in the entire range oi U a.t D = 0. 

When D and U become both finite, the system exhibits 
a variety of phases with different spin and orbital order- 
ing. Among them, the most interesting phase is the 
thrce-sublattice spin-orbital ordered state in the range of 
0-5<^<l-5and2<t/<4. This phase is insulating 
[gap is plotted in Fig. 3(c)] and nonmagnetic, appear- 
ing on the verge of the metal-insulator boundary and 
the magnetic-nonmagnetic boundary [dotted and thick 
curves in Fig. 3(a), respectively]. In other words, it is sta- 
bilized in the competing region among the paramagnetic 
metal, band insulator, and magnetic Mott insulator. 

The three-sublattice ordered state shows a trimer- 
type orbital ordering, similar to that in the previously- 
proposed trimer state in Ref. 15. The spin state is, how- 
ever, quite different from the S — 1 spin-singlet; each of 
the two tr-bond orbitals is dominantly occupied by up 
or down electron at each site, and the intersite a bond 
is formed by a pair of the spin-up and down orbitals, as 
shown in Fig. 3(b). This is different from the S = 1 high- 
spin state with the total spin (5^) ~ 2 [see Fig. 3(c)]. 

This state is distinct from that expected in the two 
well-known limits for the bond formation. One is a spin- 
Peierls type instability in the strong coupling picture, as 
discussed in Refs. 14 and 15; in this case, the spin-singlet 
bond is formed through the superexchange coupling be- 
tween the localized 5 = 1 spins. The other limit is the 
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Fig. 4. (Color online), (a) Fermi surfaces at {D,U) = (1,2) ob- 
tained by the Hartree-Fock calculation. The large hexagon repre- 
sents the first Brillouin zone. Dark grey circle and hexagon around 
the F point represent electron pockets. The light grey triangles 
around the K points represent hole pockets, (b) Corresponding 
band dispersions. Fermi energy is set to be zero. 



Peierls instability to form a bonding orbital in the weak 
coupling picture; ^^-^ here each bonding orbital is unpo- 
larized and spins do not play an important role. In our 
trimer state, the bonding orbitals are spin polarized to 
form the intersite spin-up and down pairs, whereas the 
polarization is cancelled at each site between two oc- 
cupied orbitals. This is considered to be a compromise 
between the spin-Peierls-type superexchange physics and 
the Peierls-type bonding orbital formation. 

The stabilization mechanism of the trimer-type orbital 
ordering is understood as follows. In the paramagnetic 
metal close to the metal-insulator phase boundary, one 
of the three bands makes a hole pocket of the Fermi sur- 
face around the K point, and the other two form two 
electron pockets around the F point, as exemplified in 
Fig. 4 for the case of {D,U) = (1,2). As approaching 
the metal-insulator phase boundary, the pockets shrink 
and the nesting is developed between the hole pocket and 
electron pockets. At some point under sufficient nesting, 
finite Coulomb interactions induce an instability toward 
some symmetry breaking, as often seen in spin- or charge- 
density-wave transitions. In the present case, the nesting 
is between the different orbitals, and therefore, such in- 
stability gives rise to an orbital ordering. The ordering 
is of period three because the vector connecting the F 
and K points is one third of the reciprocal lattice vector. 
Because this picture is simple and robust in the pres- 
ence of the trigonal-field splitting, we believe that the 
nesting mechanism and the resulting trimer-type orbital 
order will survive even when going beyond the mean-field 
approximation and including fiuctuation effects. 

Let us discuss our results in comparison with experi- 
ments. First of all, our nonmagnetic trimer state appears 
in the intermediately correlated region, which is in quali- 
tative agreement with experiments in LiV02. In particu- 
lar, the fact that our trimer is in between the band insula- 
tor and Mott insulator well accounts for the comparable 
energy scale between the spin and charge gap. The calcu- 
lated values of the gap, shown in Fig. 3(c), is on the or- 
der of 0.1 cV, when considering the bandwidth ~ 4 cV in 
the first-principles calculation.^'^-' This is compared favor- 
ably with the experimental estimate. ^^-^ Furthermore, our 
trimer state is on the border of the metal-insulator tran- 
sition to the paramagnetic metal; this agrees with the 



3 



J. Phys. Soc. Jpn. 



LETTERS 



experimental trend in the substituted materials LiVX2 
(X=0, S, Se).^^-' Our trimer state is nonmagnetic, but 
each occupied orbital is spin polarized under the electron 
correlation, which is clearly distinct from the low-spin 
nonmagnetic band insulator under strong trigonal field: 
It is desired to reexamine the x-ray absorption spectra, 
which was argued to be consistent with the high-spin 
state, by taking our spin-orbital ordered trimer state 
into consideration. To confirm our scenario, it is impor- 
tant to identify the band structure and the Fermi surfaces 
in the metallic compounds, such as LiVSe2, experimen- 
tally or by first-principles calculations. A quantitative 
estimate of Coulomb interactions is also helpful. It will 
be also interesting to experimentally examine the effect 
of uniaxial pressure which dominantly affects D. 

Interestingly, our phase diagram includes the square- 
type AF state [Fig. 1(b)] in the region U > 7 for 
nonzero D. The spin-orbital order coincides with that 
in the lowest-temperature phase in a related compound 
NaV02.^'^^ The compound is indeed more strongly- 
correlated than LiV02, evidenced by the larger charge 
gap and lattice constant. We, however, note that a 
LSDA+[/ calculation predicted the opposite sign of D 
for NaV02.^^-' Extension of our analysis in a wide range 
of parameters including £> < is left for future study. 

In our study, we considered the cr-bond overlap to- only. 
We confirmed that our trimer state remains robust for 
more general band structure when ta is dominant. We 
note that the value of D for our trimer state is larger 
than that obtained by the first-principles calculation.^"' 
We also note that U might be small compared to the 
estimate from fitting of the photoemission spectra for a 
related perovskite.^^' The parameter range for our trimer 
state, however, may be extended to smaller D and larger 
U region when we include the concomitant lattice distor- 
tions with trimer formation, as observed in experiments. 
We confirmed this tendency by considering the model (1) 
on an isolated three-site cluster. Meanwhile, the cluster 
calculation predicts that another magnetic trimer state 
(d) is also extended and reach D = 0. We speculate that 
this state is related with the spin-singlet trimer state ob- 
tained in the exact diagonalization for a three-site cluster 
at _D = in Rcf. 15. An interesting question is, in the 
presence of lattice modulation, how the magnetic and 
nonmagnetic trimer states are modified in more sophisti- 
cated calculations beyond the mean-field approximation. 
This problem is left for future study. 

In conclusion, we have investigated the origin of the 
trimer formation in the Hubbard model with t2g or- 
bital degeneracy on a triangular lattice. We found that 
a nonmagnetic trimer state with three-sublattice spin- 
orbital ordering is stabilized by nesting between dif- 
ferent orbitals on the border of metal-insulator transi- 
tion under the trigonal crystal field. This state takes 
the most efficient of both the spin-Peierls instability as- 
sisted by orbital ordering in the strong coupling pic- 
ture and the Peierls instability by forming bonding or- 
bitals in the weak coupling picture. Our result under- 
lies a general mechanism of the cluster formation in the 
intermediately-correlated orbital degenerate systems. It 



also potentially provides comprehensive understanding of 
a variety of the ground states in the related compounds 
UYX2 {X=0, S, Se) and NaV02. 
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